Effects of Item Order and Response Options in College Student Surveys

Given numerous pressures toward greater accountability and transparency in higher education, obtaining high-quality data about students’ experiences and outcomes ismore important than ever. A number of recent studies have examined issues related to college student surveys, including survey nonresponse (Adams & Umbach, 2013; Laguilles, Williams, & Saunders, 2011), survey completion (Barge & Gehlbach, 2012; Laguilles et al., 2011), socially desirable responding (Bowman & Hill, 2011; Gonyea & Miller, 2011), and satisficing (i.e., suboptimal cognitive processing that results in low-quality responses; Barge & Gehlbach, 2012; Chen, 2011). As a whole, these studies suggest that careful attention and inquiry is critical for drawing valid conclusions from college student surveys. This study explores two issues that have received very limited attention in higher education. First, the order in which items are presented may play a role in shaping students’ responses. For example, the Cooperative Institutional Research Program’s College Senior Survey (CIRP CSS) asked students about their own learning and growth at the beginning of the questionnaire from 2010 to 2012, whereas these items were placed differently—toward the middle of the survey—in the preceding 17 years (see Higher Education Research Institute [HERI], 2014). The National Survey of Student Engagement (NSSE) asked about self-reported gains toward the end of the questionnaire from 2003 to 2013 (appearing before the demographics), whereas these were somewhat closer to the middle of the instrument in 2000–2002 (NSSE, 2014). To what extent might these item order differences within and across surveys affect student responses? Second, the response options that are provided for a given item or set of items may also shape the results. The CIRP CSS and NSSE, along with many other surveys that are administeredwithin and across institutions, ask about the number of hours per week in which students engage in various activities